Watermark systems and methods

ABSTRACT

Various improvements relating to digital watermarking and related technologies are detailed, including methods that enhance security and functionality, and new articles including watermarked puzzles and marked DNA.

RELATED APPLICATION DATA

This application is a continuation of application Ser. No. 10/122,141,filed Apr. 12, 2002, which claims priority benefit to provisionalapplication No. 60/284,163, filed Apr. 16, 2001.

TECHNICAL FIELD

The present disclosure memorializes various improvements relating todigital watermarking and related technologies.

BACKGROUND

Digital watermarking is the science of encoding physical and electronicobjects with plural-bit digital data, in such a manner that the data isessentially hidden from human perception, yet can be recovered bycomputer analysis. In physical objects, the data may be encoded in theform of surface texturing, or printing. Such marking can be detectedfrom optical scan data, e.g., from a scanner or web cam. In electronicobjects (e.g., digital audio or imagery—including video), the data maybe encoded as slight variations in sample values. Or, if the object isrepresented in a so-called orthogonal domain (also termed“non-perceptual,” e.g., MPEG, DCT, wavelet, etc.), the data may beencoded as slight variations in quantization values or levels. Thepresent assignee's U.S. Pat. No. 6,122,403, and application Ser. No.09/503,881(now U.S. Pat. No. 6,614,914), are illustrative of certainwatermarking technologies.

Watermarking can be used to tag objects with a persistent digitalidentifier, and as such finds myriad uses. Some are in the realm ofdevice control—e.g., tagging video data with a do-not-copy flag that isrespected by compliant video recorders. (The music industry's SecureDigital Music Initiative (SDMI), and the motion picture industry's CopyProtection Technical Working Group (CPTWG), are working to establishstandards relating to watermark usage for device control.) Otherwatermark applications are in the field of copyright communication,e.g., indicating that an audio track is the property of a particularcopyright holder.

Other watermark applications encode data that serves to associate anobject with a store of related data. For example, an image watermark maycontain an index value that serves to identify a database recordspecifying (a) the owner's name; (b) contact information; (c) licenseterms and conditions, (d) copyright date, (e) whether adult content isdepicted, etc., etc. (The present assignee's MarcCentre service providessuch functionality.) Related are so-called “connected content”applications, in which a watermark in one content object (e.g., aprinted magazine article) serves to link to a related content object(e.g., a web page devoted to the same topic). The watermark canliterally encode an electronic address of the related content object,but more typically encodes an index value that identifies a databaserecord containing that address information. application Ser. No.09/571,422 details a number of connected-content applications andtechniques.

One problem that arises in many watermarking applications is that ofobject corruption. If the object is reproduced, or distorted, in somemanner such that the content presented for watermark decoding is notidentical to the object as originally watermarked, then the decodingprocess may be unable to recognize and decode the watermark. To dealwith such problems, the watermark can convey a reference signal. Thereference signal is of such a character as to permit its detection evenin the presence of relatively severe distortion. Once found, theattributes of the distorted reference signal can be used to quantify thecontent's distortion. Watermark decoding can then proceed—informed byinformation about the particular distortion present.

The assignee's application Ser. Nos. 09/503,881and 09/452,023 (now U.S.Pat. Nos. 6,614,914 and 6,408,082) detail certain reference signals, andprocessing methods, that permit such watermark decoding even in thepresence of distortion. In some image watermarking embodiments, thereference signal comprises a constellation of quasi-impulse functions inthe Fourier magnitude domain, each with pseudorandom phase. To detectand quantify the distortion, the watermark decoder converts thewatermarked image to the Fourier magnitude domain and then performs alog polar resampling of the Fourier magnitude image. A generalizedmatched filter correlates the known orientation signal with there-sampled watermarked signal to find the rotation and scale parametersproviding the highest correlation. The watermark decoder performsadditional correlation operations between the phase information of theknown orientation signal and the watermarked signal to determinetranslation parameters, which identify the origin of the watermarkmessage signal. Having determined the rotation, scale and translation ofthe watermark signal, the reader then adjusts the image data tocompensate for this distortion, and extracts the watermark messagesignal as described above.

With the foregoing by way of background, the specification next turns tothe various improvements. It will be recognized that these improvementscan typically be employed in many applications, and in variouscombinations with the subject matter of the patent documents citedherein.

DETAILED DESCRIPTION Secure Transmission of Watermark Data

In application Ser. No. 09/571,422, a client-side application (a“reader” program) processes frames of video data from a web cam or otherimage sensor, and decodes watermarked information therefrom (e.g., basedon luminance values in 128×128 pixel blocks). This information is thentransmitted to a remote server, which responds to the client with acorresponding URL or other reply. The client-side application can theninitiate a link to the specified URL, or otherwise respond to the reply.

An improved method offers enhanced security. As before, the client-sideapplication processes frames of video data, and decodes watermarkedinformation. This time, however, the application applies losslesscompression to the block(s) of luminance values from which the watermarkinformation was decoded, and may time-stamp this compressed block ofinformation. The block may then be encrypted, e.g., using a private key(which may, or may not, be part of a private/public key pair). Thisencrypted block of information is then transmitted to the remote server.

The remote server decrypts the block (if necessary), and checks the timestamp to ensure that the data was stamped within an expected priorperiod (e.g., within the past 60 seconds). The compressed block is thendecompressed, and the watermark is read from the luminance information.The remote server then determines the appropriate response for thatwatermark (e.g., a URL), and takes the corresponding action.

It will be recognized that there are a number of variations possible insuch arrangements. As noted, the time-stamping and the encryption areoptional. Lossy compression can be used instead of lossless. Or, ifbandwidth constraints are not an issue, the block(s) can be transmittedwithout compression. Or, instead of transmitting the raw luminanceinformation, it may be filtered in some manner first (e.g., medianfiltered), and the filtered data can be sent to the server(time-stamped, compressed, and/or encrypted). In some arrangements, theclient-side application does not first decode the watermark, but insteadtransmits a block of luminance data prior to any watermark decoding. Insome arrangements, the watermark information is not conveyed in theluminance data, but is conveyed otherwise.

It will be recognized that techniques such as those described above findapplicability beyond the particular context of the '422 application, andmay be used, e.g., in connection with watermarked still imagery,watermarked audio, etc., etc.

Digital Asset Management

Watermarking can play a key role in Digital Asset Management (DAM)systems.

Consider a “deep” web searching system in which a web searching agentruns locally on all registered web servers and reports back to a centraldatabase available for general web searching. To the user, the systemlooks just like Google or AltaVista. Advantages of the architectureinclude the following. The directories and times to search can becontrolled by the web server webmaster. By running locally, the searchagent can also search non-html files, such as Word documents, databasesand linked media for deep searching. By running in a distributedarchitecture, more content can be searched and categorized. The webagent could run as a distributed agent on the web server, using idlecompany computers in the evening. In addition, the searching agent isintelligent. The agent can use tools such as RuleSpace for text andVirage for video categorization.

Images, audio, and video that are watermarked can be categorized andhave associated usage rules based upon linking the watermark ID to oneor more remote database servers, such as the “Grand Central” serverdetailed in the '422 application. Watermarked web content can be bettercategorized, thus improving consumers' searches and properly indexingevery company's web server.

A similar novel structure can be used for internal digital assetmanagement (DAM). This DAM structure runs within the company's Intranet,and the web agent runs on every employee's computer. More specifically,each employee marks directories that are continually searched,categorized and reported to the central Intranet search site. Theemployee moves important documents to that directory when finished, orallows people to search on documents in process. This helps employees oflarge companies to access company information (e.g., it helps HP knowwhat HP knows).

While the above structure helps locate digital assets and associateusage rules, the system can also show the relationship betweendocuments. For example, when a document is found in a search, all of thelinked documents, such as for html, word, etc., and inserted objects,such as images, audio, video, etc., can be displayed.

By watermarking images, audio and video with IDs, the content can becategorized and associated with rules via the Grand Central database.

One of the key obstacles with any DAM system is the cost of inputtingthe metadata associated with each asset. By using watermarks to identifyand link through a server (such as the Grand Central” system), thisissue can be addressed.

Consider: a user takes a picture with a digital camera and stores theimage in a DAM system. The user enters associated metadata (maybe thename of the beach it was taken on). The image is watermarked with anImageID. The user now distributes the image to her business partners.One partner takes the image and stores it in his DAM system. This systemrecognizes the watermark, links through Grand Central to the firstuser's DAM system—which responds by supplying all the metadata. Thisdata is automatically entered into the partner's system—improvingproductivity and accuracy, and gaining metadata that could not bedetermined from the image itself (the name of the beach).

This may be regarded as a way of allowing disparate DAM systems tointeroperate.

The following article on DAM systems gives some more context to theforegoing, and illustrates some of the variety of systems in which thedetailed technology can be employed:

Watermarked Puzzles

Jigsaw puzzles offer a great variety of applications for digitalwatermark technology. To name but a few:

A puzzle can have a watermark that is readable only when the puzzle iscompletely assembled properly. When the completed watermark is sensed bya webcam or the like, and relayed to Grand Central, the user'saccomplishment can be acknowledged with a variety of “rewards” (e.g., acongratulatory message, a prize, etc.).

The rewards can be served by the remote server and delivered to theuser's computer. Or the remote server can trigger a reward that islocally stored on the user's machine. Instead of rewards, other actionscan be triggered, such as linking to different URLs.

In a variant of the foregoing, the puzzle may be designed so it can beassembled in several different ways (e.g., many of the pieces haveidentical shapes, so can be substituted for like-shaped pieces). Byassembling the puzzles in different ways, different watermark patternsare formed, and different prizes can be triggered. Or only selected waysof assembling the puzzle may trigger a prize.

The puzzle can form part of a game, including an on-line game or amulti-player game. Advancing through the game to more advanced levelsmay require demonstrating increased proficiency in assembling thepuzzle. The game may pose tests or challenges that require correctassembly of the puzzle to meet.

The puzzle may or may not be printed with conventional puzzleartwork/graphics. In the latter case, all pieces may have a generallyuniform printing pattern (e.g., a high-strength watermark pattern).

Puzzles other than jigsaw puzzles can use watermark technology as well.

Trade Shows and Wireless Data Broadcasting

Trade show booths have historically distributed printed productinformation. With the advent of wireless PDAs, new techniques ofdistributing product information become feasible.

One is to transmit image, audio, or video objects (e.g., files) topassers-by. Such content can be digitally watermarked with an ID thatallows it to link through a remote database (e.g., Grand Central). Whena visitor receives such an object, it can later be viewed on the PDA (oron another computer to which it is transferred. If the visitor wants toreceive more information, a user interface can be actuated to effect alink to an on-line resource, such as a web page. One user interface isright-clicking on the object, and selecting from a displayed menu anoption that links (e.g., through Grand Central) based on the watermarkinformation encoded in the object. A great variety of other userinterface paradigms can likewise be used.

Similar arrangements can be effected using technologies other thanwatermarks. For example, an object identifier can be stored in a fileheader, or otherwise associated with the object, and forwarded to aGrand Central-like remote server to initiate a link to an on-lineresource. Or a literal URL can be conveyed with the object—in a header,by a watermark, or otherwise.

Thermochromic Inks

There is a class of inks whose characteristics vary with temperature.Most commonly, it is the color of such inks that varies withtemperature.

Watermarks can be printed with thermochromic inks. When sensed by acolor web cam, or other image sensor device, the color of the ink can beused to trigger an appropriate response (or to trigger no response atall).

Consider an ink that looks blue at 50 degrees Fahrenheit, and red at 80degrees. Image data gathered by a color image sensor can segregate thedifferent color channels (e.g., red/green/blue) and analyze each for awatermark. If a watermark is found in the red channel, a first responsecan be triggered (e.g., through Grand Central). If a watermark is foundin the blue channel, a second—different—response can be triggered.

In the case just given, the watermark payload is constant (the inkpattern carries a single payload)—the difference arises by the colorchannel in which the watermark is sensed. This information would berelayed to Grand Central (e.g., as context information) with the payloadso that different responses can be triggered in the two different cases.

In this case, as in others, the watermark-triggered action need notinvolve a remote server such as Grand Central. Instead, the localwatermark decoder can trigger different actions locally.

As noted, only a single action may be involved, and it may be triggeredonly when the watermarked object is imaged at or near a predefinedtemperature. For example, product packaging may be watermarked with suchink as part of an anti-counterfeiting program. A suspect product may bescanned for a watermark at room temperature, and again afterrefrigerating to 50 degrees. Only if the watermark is sensed at thelower temperature (in the blue channel) and not at room temperature,does the package pass this test.

In application No. 60/257,822, filed Dec. 21, 2000, the use of UV and IRinks in watermarking was discussed. Other disclosure on this topic isincluded in application Ser. No. 09/562,516.

It should be noted that such techniques are particularly well suited indeterring counterfeiting, e.g., product packaging, value documents, etc.Such markings are difficult to identify and reproduce. Handheld scanningdevices can include UV or IR illumination sources, and read a watermarkon a scanned object.

In preliminary studies, green, yellow, and red fluorescing UV inks seemto perform better than blue fluorescing UV ink.

Three types of IR inks are particularly contemplated for suchapplications: IR Fluorescing Ink: Produces an invisible printed imagethat vibrantly fluoresces red when illuminated in short wave UV blacklight and a much less weaker red in long wave UV black light.

IR Dual Fluorescent Ink: Produces an invisible printed image thatfluoresces in one color when illuminated with long wave UV black lightand in an entirely different color when illuminated with short wave UVblack light.

IR Invisible Readable Ink: Produces a generally invisible printed imagethat is identified in a very specific band of infrared spectral range.

The UV or IR inking can be applied by adding one additional plate to aprinting run. (An additional color separation may be required in someprint applications.)

Printing on a uniform background generally gives better readingreliability results than printing on a picture-printed background, withlighter uniform backgrounds seeming to perform better than darkerbackgrounds.

Banknote Security with Proteins and Biological Materials

One options for a banknote security feature involves placing proteins onthe banknote which are then detected by a scanning device. In oneembodiment, a banknote printer or issuer would pick an area of the noteand place a few proteins or other biological identifiers within or ontop of the substrate. The proteins would be very difficult if notimpossible to reproduce unless a counterfeiter had the correct“encoding” device. These proteins could fit within a sub-micron space onthe note or be spread all over the surface, like a digital watermark.The proteins may be, e.g., less than a thousandth of a micron in size,and may fluoresce (monochromatically, or with different colors) whensubject to certain irradiation.

A scanning device would detect the existence of the proteins—if theproper proteins are detected then the note is legitimate. The scanningdevice could use any number of methods to find detection. The proteinscould fluoresce or the proteins could have some other unique code orpattern that the scanner identifies. The device could be a visual cue orthe protein could fluoresce a certain way only under certain lightingand temperature conditions such as infrared lighting. Using otheridentifier methods the protein identifier could be unique to the printrun or denomination, and there could be unique identifiers to the actualindividual note.

The proteins could be placed on a “tab” that is then placed on the note,similar to how holograms are bonded, or alternatively the proteindelivery system could place the proteins directly on the substrate. Theproteins have to survive many stresses of light, bacteria, fingers,washing etc.

The same techniques can naturally be used on value documents other thanbanknotes (e.g., passports, financial instruments, etc).

Watermarks in Electronically Displayed Data

As noted in application No. 60/257,822, a watermarked image can bepresented on the

LCD display of a wristwatch or the like, and captured by a webcam forvarious purposes.

It should be noted that the display can present alphanumeric text, whichcan then be decoded from the webcam image using known optical characterrecognition (OCR) techniques. Linking and other operations based on thisinformation can then be undertaken, as described in the cited and '422applications.

Rotating URLs for Security

Imagine a company or organization wants to have a site accessiblethrough Digimarc MediaBridge (via the Grand Central server), but wantssome level of security for the site. That is, they don't want links tothe site emailed around, they don't want search engines indexing thesite, etc. For example, Integraf may send out a brochure with “exclusivesecure” access to a Intergraf 2001 exhibitors web site. The site hassome long obscure URL such as

www<dot>trytotypeinthislonnngobscureurlwithoutmakingamistake01112344567<dot>com.The brochure, of course, is watermarked with an ID that links to thesite (through Grand Central). Now imagine that the URL actually changesperiodically. Grand Central knows of the change so that the brochurecontinues to serve its function, but saved links, forwarded links andsearch engine indexes become rapidly obsolete. The security of the siteis controlled by the life of a URL. It could be varied from one secondto one week depending on how sensitive the owning organization is. Tothe holder of the brochure (or other access document), the change istransparent, since the GrandCentral database changes the destination URLin synchrony with its movement by the organization.

A generalization of all of this is that each and every object can haveits own, generally inaccessible, rich source of attached information andinherent interactivity. This could be an entire platform all to itself.

More on Watermarks and Handheld Detectors for Detection of Counterfeits

This process assumes that a product package is watermarked, or that aproduct has a watermarked hang tag. A handheld detector, such as a PDAwith image capture (PalmPilot-like), would capture an image of thepackage/tag. Then:

-   -   The image would be examined for the presence of a watermark and        would indicate on the device the presence or absence of the        watermark. Or    -   The image would be transmitted via a wireless link to a central        site where the presence or absence of a watermark would be        determined and an appropriate response returned to the handheld        for display. Or    -   If the transmission of the image to the central site was        unsuccessful, either because of a response time constraint or        the lack of available transmission medium, the detection would        be performed locally as in the first situation.

Watermarks and Collectibles

Watermarks find various applications in connection with collectibles,action figures, and the like. Some of these are detailed in copendingapplication Ser. No. 09/630,243 (particularly focusing on baseballcards).

Action figures have expanded beyond G.I. Joes, and now include a rangeencompassing:

-   KISS action toys-   X-Files-   Army of Darkness-   Janice Joplin-   Edward Scissorhands-   Doug and Bob MacKenzie-   Spawn

Many collectors buy two of an item—one to play with and one to keepshrink wrapped. Watermarks can be applied to the toy itself (e.g., thetoy base) or to the packaging. The functionality served by the watermarkcan include confirming authenticity, linking to associated internetsites, serialization, etc.

Take, as an example, the Sanrio family of character branded merchandiseavailable, such as Hello Kitty (see www<dot>sanrio<dot>com). Bywatermarking products to permit Digimarc MediaBridge brand linking bycustomers, the brand manager can obtain real-time information aboutmarket acceptance of each different product, including which productsare the most popular, the correlation between an ad campaign and salesof a product, etc. (Previously, the brand manager needed to wait forsuch information to filter back up the retail supply channel, preventingreal-time response strategies.) The purchasers, in turn, could beprovided with opportunities to win prizes, participate in games, learninformation, register for club benefits, etc., by linking from theproducts. The linking and interactivity provided by such watermarkfunctionality may contribute to the cachet of some such products.

Watermarks and Cookbooks/How-To Books

Books, magazines, and other publications can be watermarked to permitusers to link to on-line communities (e.g., discussion groups or forums)of other purchasers of the same items. The publication can be marked inits entirety, or just a cover, or just an internal section, etc.Different sections can have different marks and link to differenton-line communities.

Thus, a home improvement book about building decks can permit apurchaser to link-to/correspond-with other individuals engaged in thesame undertaking (and possibly link to the book's authors). The on-linecommunity can be further tailored by the user's geographic location(which can be indicated to the remote server computer, e.g., bytransmitting the user's zip code together with watermark information).Regional chat about the book or its subject can thus be facilitated(e.g., a reader in rainy Oregon may post a message asking others in thearea about waterproofing treatments that have been used with success intheir shared climate).

Cookbooks are also well suited to such techniques—providing forumslinking cooks with like-minded interests (e.g., purchasers of a tofucookbook living along the Gulf coast). Again, different chapters canlink to different communities.

The same approach of defining an on-line community of owners of a givenpublication can be extended beyond publications to any item (e.g.,collectors of G.I. Joes <linking from product or packaging>, FordExplorers <linking from watermarked key>, etc.).

Soft Bit Errors and Fragile Watermarking

Fragile watermarks are known, e.g., in pending application Ser. No.09/433,104 (now U.S. Pat. No. 6,636,615).

Content (e.g., audio or imagery) can be marked with a watermark, anddecay of the watermark through compression and other processing can beused to determine the quality of the content. For example, the number ofbit errors in the payload (including CRC and false positive bits) coulddetermine the quality.

One procedure for measuring such decay is as follows:

-   -   1. Use the payload read from the watermark to re-create the        original embedded bit sequence (including redundant bits) used        for the watermark.    -   2. Convert the original bit sequence so that a zero is        represented by −1 and a one is represented by 1.    -   3. Multiply (element-wise) the soft-valued bit sequence used to        decode the watermark by the sequence of step 3.    -   4. Create two measures of watermark strength from the sequence        resulting in the previous step. The first measure is the sum of        the squares of the values in the sequence. The second measure is        the square of the sum of the values in the sequence.    -   5. These two measures can be combined in various fashions to        yield a final metric, or can be used individually in assessing        watermark decay.

Watermarking Descriptors

The header of a file, such as the header of a file storing compressedvideo, e.g., in

MPEG format, can include data conveying information about the form ofwatermark used in the file contents. The information can be literallyexpressed in the header, or information in the header can serve as alink to a remote data repository at which information about the filewatermark is stored.

One approach is to employ XML tags in the header (MPEG-7 uses XML),e.g.:

-   -   <Watermarking Type>    -   <Watermarking Message>

A dictionary would desirably be established for each term. For example,<Watermarking Type>could include Digimarc Image version 1, DigimarcImage v2, Digimarc Video v1, Digimarc Video v2, Digimarc Audio v1,Digimarc Audio v2, Philips Video v1, Philips Video v2, etc. (includingevery existing watermarking company). And <Watermarking Message>couldrepresent the watermark payload in a known format.

For example, application message type 4 detailed in application No.60/256,628, e.g.:

Conten- # of Owner Content- Distributor Message Info CMC WM Content-Owner WM Distributor Distributor Misc Type Bits A/V Bits Version OwnerID Object ID Version ID Object ID Info 16 bits 16 bits 1 bit 31 bits 8bits 96 bits 96 bits 8 bits 96 bits 96 bits 96 bits

These descriptors are beneficial since there may be many watermarksembedded in the content and it can take too long for the end-user whilethe system decodes all of the potential watermarks. A benefit of theheader descriptors is that they increase the computational efficiency ofreading watermarks since the watermarks only need to be read once, andthen can be added to the MPEG-7 description (if using both descriptors).

At least, the types of watermarks will be known and all watermarks don'tneed to be searched if the watermark message descriptor is not added forsecurity reasons, although protected by MPEG-7 IPMP. If the meta datainformation contained in the header is suspect, then the information canbe retrieved from the watermarked content itself. Likewise, if the metadata information in the header becomes lost, it can be regenerated fromdata conveyed by the content itself.

(Redundant representation of information in both the header and acontent watermark was more generally disclosed in a series of patentsfiled by the present assigned in May, 1995, including U.S. Pat. Nos.5,748,763, 5,850,481, 5,748,783, and 5,636,292.)

Theatre Tickets, Etc.

Movie and other event tickets may be purchased on-line from varioussources. In this implementation, however, the ticket purchaser printsthe purchased tickets on her home computer system (and printer). Theprinted tickets include embedded watermark data. (A ticket image, havinga unique identifier or purchase code embedded therein, could betransferred to the user's computer for printing. Alternatively, theonline movie ticket retailer transmits a payload or an authenticationcode to the user's computer. A plug-in is launched, which incorporatesthe payload information when creating and printing the tickets.).

At the movie theater, the ticket purchaser presents the watermarkedticket to a decoder. The decoder verifies authentic tickets by opening agate or enabling a visual confirmation, e.g., a green light. The movietheater decoder can download a list of authentic payloads or identifiersprior to each showing, or may query an online database to verify eachticket. (Fragile watermarks are alternatively embedded in the printedticket to help avoid counterfeiting.)

In still other arrangements, the user may have a talisman, such as adriver's license or key fob, that has a watermark embedded therein. Whenpurchasing a ticket, the user may present the talisman to a camera orother detection device associated with their computer system. The camerareads the watermark, and relays it to the ticket vendor to associatethat watermark payload with a virtual ticket. When the user arrives atthe theatre, they can present the same talisman for sensing. Thetheatre's computer decodes the watermark, checks the payload against alist of authorized entrants, and permits entry if the user is found tobe authorized.

Credit at a concession stand (or coupons for such) can be obtained bytechniques like those above.

Every movie theater (and sporting arena, music concert venue, etc.)should be so enabled.

Collaborative Work Environments, Etc.

Various systems permit several people to collaborate on-line on a singleproject (e.g., a document) from remote locations using shared tools. Oneis www,sharethis.com. Watermarking can be employed advantageously insuch arrangements.

Watermarking is usually the last step in a content-creation process, toensure that the watermark is not accidentally destroyed. Thus, ifcontent in a collaborative work system is not being modified but onlyshared, it could be watermarked and tracked through the system,including additional edge checks that users have the correct usagerights. Prior Digimarc applications have discussed usage scenarios likethis with respect to digital asset management, content tracking forbroadcasters, and Napster file sharing applications.

A different circumstance arises, however, if the collaborative worksystem is used to create or modify content.

In such a system, a watermark embedding function, such as a Postscriptcommand with ID and robustness fields (parameters), could be specifiedduring the creation process by the content creators, and the watermarkcould be actually embedded during rendering. For example, when designingDigimarc MediaBridge enabled packaging, the graphic designer and contentowner could define the watermarking ID and robustness since they workclosely during this stage. Then, the watermark ID is embedded wheneverviewed on the computer screen, or, more importantly, proofed and printedat the RIP. Since the embedding process can be modified at the RIP, thecolor guru that controls the RIP can make sure the watermark is robustbut invisible and the watermark can be embedded dependent upon the typeof printer.

This example demonstrates several advantages. Some advantages relate tothe fact that the watermark definition and embedding are separate. Thecontent creator (such as graphic designer) can work with the contentowner to define the watermark, while the watermark is rendered by therendering expert, such as the color guru at the RIP or audio masteringengineer. In addition, the embedding engine knows the rendering deviceand can adjust the watermark for its characteristics. Other advantagesinclude that the watermark can be embedded into structured content, suchas vector graphics, MIDI and animation.

Extending the watermarking function concept to shared work spaces, thewatermark can be added whenever the content is rendered by any client.The ID (payload), which can link to the internal representation of thecontent in the shared environment and/or the content owner, can be sentwith the content, and the rendering client embeds the ID. The renderingclient could embed its user's ID in addition to the content ID orinstead of the content ID. As such, any rendering of the content istraceable.

This watermarking function may not explicitly be passed to the clientfor embedding, but implicitly known by the client.

For example, while the content owner is demonstrating a song-in-progressto critics, the content and rendering client IDs are embedded such thatevery rendition is traceable. Thus, if a critic uses their sound card orconnected recorder to capture the song and the song shows up elsewhere,such as on Napster, the song can be traced to the client that renderedthe content as well as back to the content owner for legitimatepurchase—even though the song may have been changed after the demo.

Another important capability enabled by such approaches is variable dataencoding. Variable data refers to systems, like serialization systems,in which several copies of a content item are being produced, and eachis to have a unique (or customized) watermark ID. Often, this ID is notassigned until the moment the item is finally rendered (e.g., an objectprinted, or a CD pressed or played).

In summary, the above systems, in general, show that there arecircumstances where the embedding should be moved to the edge of thenetwork. Along similar lines, there have been other Digimarc patentfilings about moving detection out to the edges of a network, such as ona set top box. Such approaches can also be employed in conjunction,e.g., with DVD, CD, and other media recording devices that embedidentification information as content is being stored.

(An application related to the foregoing is Ser. No. 09/810,000.)

Promotional Content Distribution

application Ser. No. 09/476,686 discloses a device that listens toambient sound, and decodes a watermark from it (the device may be a cellphone, or a dedicated unit). The watermark can be used for variouspurposes, e.g., to identify a song. This technology, and such devices,are referred to by the name BirdDawg.

BirdDawg arrangements can be employed to offer promotional music. Aexample usage model is that after the user clicks on a “song info”button on the device, the central database returns to the cell phone theartist, song and album information as well as whether there ispromotional music, concert information and purchases available. If theuser selects the promotional music element (possibly one menu down undera “more info” menu), dependent upon the user's preferences, a link tothe song can be emailed to the user or the user could have an audiolocker to which the song becomes available, potentially the audio lockercould be a central system or a local system. If the music is downloaded,it could be tracked via various one-to-one promotional systems.

One promotional system provides content (e.g., music) in a format thatallows a first portion (e.g., 30 seconds) to be played freely. After 30seconds, the user is invited to download software that may permit thefull content to be played without interruption. The downloaded softwareincludes digital rights management technology, permitting the contentowner to specify conditions or limitations for use. The content may bewatermarked. An operating system-level watermark detector can look forthe watermark and interrupt the playing after 30 seconds unlessover-ridden by instructions from the downloaded software. (Ergo, if thedowwnloaded software is not present, playback stops.) Thus, the birddawgdevice triggers delivery of a promotional excerpt of the music, whichthe recipient can render fully functional by downloading (if not alreadydownloaded) certain ancillary software.

A BirdDawg device can be coupled to a car's navigation system. When auser purchases concert tickets through operation of the BirdDawgfunctionality, the system could offer directions to the concert. Thesedirections could be saved in memory or as a bookmark so they can be usedif the concert is not at the current time of the ticket purchase (whichis likely).

Maps and Geo-Watermarking Background

(This subject matter is related to that disclosed in application Ser.Nos. 09/800,093 and 09/833,013, now U.S. Pat. Nos. 7,061,510 and7,249,257.)

Digital watermarking has long been presented as a potential centralelement in digital asset management, particularly when those “assets”are photographs (implicitly including “digital images”). Copyrightlabeling, active copyright communication, marketing links, etc. and soforth, have all been well explored.

Within the universe of subject matter for photography is what is broadlyreferred to as remote sensing. Let us imagine that this includes alltypes of photography which somehow images the Earth's surface or itslandscape. Add to this class all photography which somehow has an innateconnection to a location on the Earth, and let us call thisgeoreferenced photography for lack of an imagination. In the finalanalysis, virtually all photographs one way or another have innategeographic properties, if one stretches the definition far enough (evenpurely synthetic images are created by an author existing “somewhere”).But this is an academic extreme. What's more relevant to this disclosureis that “most” photographs, including swept-scan satellite imagery andradar, also including vacation snaps at Niagara Falls, can be describedas having innate, if not always explicit, geographic properties. “Time”should also be included in these properties. The march of technologicalprogress is transitioning more and more photography from the “innate”category to the “explicit” category through the use of GPS technologyand/or local wireless technologies.

This disclosure concentrates on how digital watermarking (andspecifically, its database linking properties) and georeferencedphotography might inter-relate. The goal is to explore how the coreutility of the former can be used as a platform to simplify andtransform the latter. New capabilities would hopefully emerge in theprocess and from the result.

Details

It is well known and well explored that virtually all naturally takenimages can be referenced by a “6+1” dimensional vector relative to theEarth's coordinate system. The six initial elements, in one givenscheme, include:

-   Latitude-   Longitude-   Height (as compared to a mean-sea level sphere with an arbitrary    time origin)-   Time-   Cardinal Direction-   Azimuth

The “extra” dimension is itself multi-dimensional in nature,representing “sensor geometry”, where there are a variety of types, eachrequiring various rules on how it is defined and how it affects theprevious six parameters. Critical as these particulars might be for manyapplications, they are secondary to this disclosure. Suffice it to saythat a simple rectangular fan or pyramid centered on a camera's apertureis the most common form of sensor geometry and can be used as a stand-infor many others. But as one final academic point, however, the notion ofthe “sensor geometry” is simply vernacular for a coherent set of opticalsampling functions corresponding to each pixel and/or microdensityregion of a photograph.

Beginning with the now-mature area of remote sensing, but extending toall photography with an innate 6+1 dimensional geovector as describedabove, digital watermarking itself can be extended to embrace thisfundamental set of information inherent in each and every photograph. Asthe “copyright” is fundamentally a part of each and every photograph, sotoo is the “geovector” (if we can call it that, including time) afundamental part of every photograph, and digital watermarking canexpressly contain this information. As with the large prior art ofdigital watermarking explains, this information can either be containedin the embedded watermark information itself, or contained in a databaseto which the watermark represents a pointer, or both. Furthermore, allof the comments and explanations of redundant header structures applyhere as well; in other words, certain geovector information might betriply redundant:

watermark payload, header, database.

Standardization efforts are currently underway which are extending theidea of the geovector well beyond the basic elements presented above.Indeed, the above description is pretentious in its brevity relative tothese efforts. See for example [the geospatial and GIS efforts; thedigital earth, whatnot, all having simple coordinate systems at theircore; see also vvwvv.opengis.org]. All of these efforts lend themselvesto digital watermarking payloads, classic header structures, andpointed-to elements in an associated database.

A natural question to ask at this point would be: why? Why do all thiswith watermarking, won't standardized header structures work just fine?

Indeed headers alone would work just fine, in the abstract. Imagining aworld where all things are digital and all header files stay permanentlyattached to their associated image data, watermarks are unnecessary. Butin today's world, it is precisely this abstract property of permanentattachment which the digital watermark provides. This is the brain-deadfirst reason for including digital watermarking in the puzzle.

There is another reason digital watermarks might be a pragmaticfoundation for a massively georeferenced system of imagery. At the endof the day, this second reason may be much more powerful that the“permanent attachment” property. The digital watermarking of photographycurrently involves the simple step of identifying an image followed hardupon by the storing of that identification in some database or acrosssome group of databases. In other words, de facto standardization ofidentifying individual imagery is already underway in the form ofdigital watermarking. All manner of digital images, photography, fileformats, prints, and whatnot, are all being registered in a singlecoherent cross-referenceable database. These aspects of digitalwatermarking are well explored in the prior art. So why not exploit thistrend while further adding the dimension of geovector information in theprocess. The result is a database or set of coordinated databases whichrepresent a searchable database suitable for geographically basedqueries. Whereas many of such systems have been around for some time nowusing classic header structures with matching database fields, digitalwatermarking possibly presents a more fundamental foundation capable ofsynthesizing past, present, and future initiatives. Time will tell.There are quite legitimate “proprietary database” concerns buried inthis notion, but in the final analysis, the only issue is ensuring acollision-free serial numbering system for identifying imagery, owners,and attributes, a task where digital watermarking is the de facto frontrunner with no runner's up in sight.

A third reason that digital watermarks should be considered in creatinga georeferenced system of images touches upon basic common sense. Aclassic notion in most standardizations across all industries is thenotion of the “stamp” or “seal” or similar concept of indicating thatsome object has successfully completed its appointed rounds of dottingi′s and crossing t′s. Call it branding, call it formality, call it asoft form of “authenticity”; the historical momentum behind such aconcept is huge. In the case of ensuring that a given image is properlygeoreferenced by whatever standards are chosen, wouldn't it be nice ifdigitally watermarking that image as a kind of final step represents aformalized good housekeeping seal of approval. Various software andhardware taught to deal with such imagery can be programmed to routinelyread these digital watermarks and display the appropriate brand logos,seals, certificates, or dancing regal elephants. Prior art digitalwatermarking disclosures explore the range of creative options (oftentargeting “branding” as a marketing concept) better than we can tryhere. In summary, digital watermarking can not only serve this commonsense function, but the “seal” itself is a functional element of thestandardization process, serving many functions including permanentattachment to the standardized and dynamic metadata.

A fourth reason that digital watermarks can be part of a georeferenceddatabase system is also a practical one: Images by their very nature canbe inter-processed, merged, split, cut up, etc. and so forth asdescribed quite fully in the prior art. This tendency is especiallyapplicable to various geo-referenced imagery applications where variousdata sets are merged and viewed as derivative images. Ask any databaseengineer or operator to manage the behind-the-scenes management ofkeeping track of the bits and pieces, and you'll quickly hear either agrown, or see a grin when they hand you the development budget for sucha system. Digital watermarks, in many if not all such applications, canbecome a good way of coordinating and keeping track of highly diverseimage components.

Encoded DNA

DNA may be tailored to convey digital information.

As is well understood, DNA is a polymer in the form of double-helix—aspiral comprising two long chains of monomer nucleotides wound abouteach other. The nucleotides each comprises a deoxyribose sugar moleculeattached to a phosphate group and one of four nitrogenous bases:adenine, guanine, cytosine and thymine. The strands are linked to eachother by hydrogen bonds between the bases, which uniquely pair: adeninewith thymine; guanine with cytosine.

As presently understood, some of the DNA components are inactive. Thatis, they can be changed essentially without consequence. As such, theymay be tailored in a manner to convey data.

In a simple application, an adenine-guanine (AG) pair may represent adigital “1”, and a cytosine-thymine (C-T) pair may represent a digital“0.” Inactive parts of an organism's DNA may thus be tailored so thatthese inactive components serve to convey digital data. This hasnumerous applications, including forensic tracking (e.g., uniquelymarking different strains of anthrax).

Desirably, data encoded in DNA is encoded redundantly, so thatcorruption of some part of the structure does not cause data loss. Thesame data may be represented at several different locations in the DNAstructure. Or, sometimes more desirably, error-correcting codingtechniques, such as BCH (“trellis”), convolutional coding, and turbocodes, can be employed so that the correct data payload can be discernednotwithstanding sometimes severe corruption of the structure.

The data conveyed by DNA need not be a single digital string (e.g.,representing a number), but may represent several different types ofdata, e.g., an index number, a creation date, a proprietor, etc. Theindex number can serve to identify a database record containing moreinformation associated with that DNA. The data can also compriseexecutable software code or other instructions.

In addition to conveying data, the inactive components of the DNA canalso serve as synchronization markers, e.g., indicating where encodeddata starts or stops.

The data needn't solely be represented by the pattern of inactivecomponents. In some arrangements, use can be made of the activecomponents as well. For example, an inactive component in a range thatalso includes active components can change some statistic or attributeof the range (e.g., changing the number of A-G pairings to an evennumber may represent a “1;” changing the number to an odd number mayrepresent a “0”). Different ranges of the DNA structure may conveydifferent parts of the payload.

It will be recognized that DNA is susceptible to conveyance ofinformation by forms of expression other than binary. In the examplejust given, base 4 representations may be used:

Value # of A-G Pairings # of C-T Pairings 0 Even Even 1 Even Odd 2 OddEven 3 Odd Odd

If A-T pairings are distinguished from T-A pairings (and C-T pairingsdistinguished from T-C pairings), then base-8 forms of expression may beused.

In other arrangements, the values of the payload encoded in the inactivecomponents can be related to, or based on, the details of the activecomponents. By such arrangement, the integrity of the auxiliary data canbe checked to ensure that it corresponds in the expected manner with theactive components.

Detection of such coding can be performed in various manners. One is byinspection techniques. Another is by gene sequencing techniques. Anotheris by de-linking the two nucleotides, and attempting to link them todifferent reference nucleotides—each expressing a different payload.Other decoding techniques may of course also be practiced.

By encoding a known pattern into the inactive components, it is possibleto gain insight into the number of replications the DNA has undergonebetween encoding and decoding. The statistics by which errors areintroduced through DNA replication can be empirically determined, orstatistically estimated. If the original DNA structure is known, thenexamination of a later generation of that structure—and assessment ofthe number of errors introduced since encoding—can allow estimation ofthe number of generations-removed that the tested DNA is from the DNAoriginally encoded.

Recap

To review, according to one aspect, the technology includes a methodthat comprises: receiving data representing a content object; processingthe content object; sending the processed content object to a remotecomputer; and further processing the content object on the remotecomputer, said further processing including decoding plural-bit datasteganographically encoded therein.

In a variant of the foregoing, the method can include time-stamping orencrypting the more compact representation of the content object sent tothe remote computer.

CONCLUSION

To provide a comprehensive disclosure without unduly lengthening thisspecification, the patents and applications cited above are incorporatedherein by references.

Having described and illustrated the subject technologies with referenceto illustrative embodiments, it should be recognized that the technologyis not so limited.

For example, while the detailed description focused on digitalwatermarks to convey auxiliary information with audio and video content,other techniques can be used as well (e.g., VBI, digital fingerprints,header meta data, etc.). Likewise, in embodiments relating to marking ofphysical objects, other machine-readable data representations can beemployed (e.g., bar codes, glyphs, RF IDs, mag stripes, smart cardtechnology, etc.).

The implementation of the functionality described above (includingwatermark decoding) is straightforward to artisans in the field, andthus not further belabored here. Conventionally, such technology isimplemented by suitable software, stored in long term memory (e.g.,disk, ROM, etc.), and transferred to temporary memory (e.g., RAM) forexecution on an associated CPU. In other implementations, thefunctionality can be achieved by dedicated hardware, or by a combinationof hardware and software. Reprogrammable logic, including FPGAs, canadvantageously be employed in certain implementations.

It should be recognized that the particular combinations of elements andfeatures in the above-detailed embodiments are exemplary only; theinterchanging and substitution of these teachings with other teachingsin this and the incorporated-by-reference patents/applications are alsocontemplated.

1. A method comprising: receiving a steganographically encoded content object; at a first processing device at a first location, processing the received steganographically-encoded content object a first time to yield a processed content object; sending the processed content object to a second processing device at a second, remote location; and further processing the processed content object on the second processing device, said further processing including decoding plural-bit data steganographically encoded therein.
 2. The method of claim 1 in which the processing of the content object the first time includes generating a more compact representation of the content object, and the method further includes time-stamping or encrypting the more compact representation of the content object sent to the second processing device.
 3. A method useful with a network of computers, comprising: employing an agent process to process files on computers of said network; decoding watermarks from file objects encountered by said agent; by reference to said watermarks, accessing metadata associated with said file objects; and collecting said metadata in a data structure useful for searching.
 4. A puzzle comprising plural distinct pieces that fit together, characterized in that the puzzle, when assembled includes a digital watermark pattern that, when sensed with a compliant reader device, triggers an action responsive to a plural-bit payload conveyed by said pattern.
 5. A method comprising: receiving a wireless transmission of a content object on a user's wireless computer, the wireless transmission being sent by a transmitter device at a trade show; within the wireless computer, decoding a watermark from the received content object; and establishing communication between the user's wireless computer and a remote computer in accordance with plural-bit data obtained from said decoded watermark.
 6. A substrate having printed thereon a pattern including a steganographic pattern, characterized in that the steganographic pattern is printed in a thermochromic ink, wherein the pattern presented thereby changes color in accordance with temperature.
 7. A printed package having a steganographic watermark pattern printed, on a portion thereof having a uniformly light background, with an ink that is transparent at visible light wavelengths, but is readily detectable if illuminated with invisible light, the pattern conveying plural bits of digital data.
 8. A banknote marked with a biological protein, the protein serving as a taggant by which the banknote can be identified.
 9. A strand of DNA having inactive components thereof deliberately configured so as to represent numeric data according to a code.
 10. A method comprising: (a) storing in a database a URL associated with an index number, the URL corresponding to the address of a networked computer on which a web page associated with the index number can be found; (b) occasionally and automatically changing the URL associated in said database with said index number; and (c) providing said web page at said changed URLs.
 11. A method comprising: capturing image data from a product or package using a portable device; transmitting data corresponding to said image data to a remote computer for processing; and if the remote computer failed to respond in an expected manner, decoding a digital watermark from the image data using said portable device.
 12. A method comprising: encoding a product or package with a digital watermark distributing the encoded item through retail channels; receiving data reporting on electronic linking performed by customers presenting the item to compliant computer devices; wherein a product manufacturer can receive prompt feedback on sales of the product.
 13. A method comprising: encoding a plural printed publications with digital watermark data; and forming an on-line community comprised of users who have custody of the publications, as evidenced by their presentation of the publications to compliant watermark reading devices.
 14. A method comprising: decoding an N-bit payload from a steganographically-encoded content object, the payload being redundantly represented in the content object by a first set of M-bits, where M>N, so as to permit correct recovery of the N-bit payload notwithstanding corruption of certain of the first set of M bits; and from the decoded N-bit payload, generating a second set of M-bits corresponding thereto, the second sets redundantly representing the N-bit payload without corruption.
 15. The method of claim 14 that further includes processing the first and second sets of M-bits to yield an indication of the corruption of the content signal.
 16. The method of claim 14 that includes transforming the first and second sets of M-bits from {0,1} symbols to {−1,1} symbols, and performing a bit-wise multiplication between the two transformed sets.
 17. A method comprising: steganographically encoding plural-bit payload data into a portion of a file representing audio or imagery, the steganographic encoding employing a watermarking protocol; and storing in the header of the file a tag indicating the watermarking protocol used for said encoding.
 18. A method comprising: at a first site, in connection with making an electronic payment for a product or service, presenting to a first image sensor a digitally watermarked talisman that conveys a payload corresponding to a user; at a second site remote from the first, presenting the same talisman to a second image sensor; and by reference to watermark payloads decoded from image data from said first and second sensors, determining whether the user has paid for the product or service.
 19. The method of claim 1 that includes applying a compression algorithm to at least some of the received content object at the first location, sending compressed data away from the first location, and decompressing said compressed data prior to decoding the plural-bit data therefrom.
 20. The method of claim 1 in which said processing of the content object the first time includes associating time data therewith and, at a location away from the first location, making a decision based on said time data.
 21. The method of claim 20 in which said decision includes checking the associated time data to determine if such time data indicates a time within an expected prior period.
 22. The method of claim 1 wherein said content object comprises audio data.
 23. The method of claim 1 wherein said content object comprises still image data.
 24. The method of claim 1 wherein said content object comprises video data.
 25. A method comprising: capturing steganographically encoded content using a first device, said first device comprising a cell phone; in said first device, processing the steganographically-encoded content a first time to yield processed content data; sending the processed content data to a second processing device at a second, remote location; and further processing the processed content data on the second processing device, said further processing including decoding plural-bit data steganographically encoded therein.
 26. The method of claim 25, in which said content data comprises image data.
 27. The method of claim 25 in which said content data comprises audio data. 